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I.  THE  PRIORITIZATION  PROCESS 


BACKGROUND 

In  May  of  1984,  SRA  Technologies,  Inc.  completed  the  first  phase  of  a  pro¬ 
ject  monitored  by  the  Office  of  Naval  Research;  the  report  on  that  project 
was  entitled  "Roadmap  for  Navy  Civilian  Personnel  Research."  The  project 
was  designed  and  implemented  in  order  to  respond  to  the  needs  of  the  Navy 
for  a  research  plan  to  gather  systematic  information  on  the  large  civilian 
personnel  workforce  which  supports  the  Navy  mission.  The  report  consisted 
of  a  series  of  research  areas,  arrayed  Into  logical  linear  sequences  de¬ 
signed  to  support  specific  mission  objectives.  In  addition,  each  research 
area  Included  a  discussion  of  the  context  of  the  research  and  a  descriptive 
summary  statement  of  the  existing  literature  in  the  area.  In  addition,  the 
report  included  a  detailed  procedure  for  prioritizing  the  accumulated  re¬ 
search  suggestions. 

Following  the  completion  of  the  Roadmap  document,  SRA  staff  began  the  pro¬ 
cess  of  prioritizing  those  research  areas  and  projects  developed  during  the 
first  phase. 


PRIORITIZATION  OVERVIEW 

The  Roadmap  document  provided  a  comprehensive  overview  of  potential  civil¬ 
ian  personnel  research  areas.  It  did  not,  however,  define  specific  re¬ 
search  projects  which  could  be  conducted  In  each  area,  nor  did  it  indicate 
which  areas  were  of  greatest  concern  to  the  civilian  personnel  community. 

In  order  to  transform  the  Roadmap  from  what  was  essentially  a  reference 
document  into  a  concrete  plan  to  guide  the  allocation  of  scarce  resources. 


e.g.  research  and  development  and  studies  monies,  a  number  of  tasks  were 
carried  out: 

1.  Working  from  the  Roadmap,  generic  categories  of  research  areas  and 
research  projects  were  Identified.  These  research  areas  and  pro¬ 
jects  Incorporated  the  original  research  concepts  but  cross-cut  the 
original  research  arrays  which  had  been  organized  by  mission  objec¬ 
tives,  e.g.  recruitment,  retention,  productivity.  The  19  research 
areas  established  for  the  prioritization  process  contained  elements 
of  multiple  objectives,  as  did  the  specific  research  projects  sug¬ 
gested  for  each  area. 

2.  It  was  decided  that  an  Initial  prioritizing  of  research  areas  and 
projects  should  be  done  by  the  forty  participants  who  had  original¬ 
ly  been  Interviewed  for  the  Roadmap,  as  well  as  some  additional  in¬ 
dividuals  knowledgable  about  civilian  personnel  Issues.  The  second 
and  final  prioritizing  was  to  be  done  by  the  Project  Guidance  Team 
(PGT) ,  made  up  of  representatives  from  OP-14,  0P-01,  the  Naval 
Material  Command,  the  Navy  Personnel  and  Research  and  Development 
Center,  and  the  Office  of  Naval  Research. 

3.  A  mall  questionnaire  was  developed  (see  Appendix  A)  and  sent  to  51 
potential  respondents.  They  were  asked  to  evaluate  each  of  the  19 
research  areas  on  the  basis  of  three  criteria: 

•  The  relative  importance  to  the  Navy  of  Improving  effectiveness 
in  a  particular  area; 

•  The  relative  need  for  new  information  in  order  to  improve  capa¬ 
bilities  within  that  area;  and 

•  The  likelihood  of  being  able  to  use  research  findings  effective- 


Each  research  area  was  then  rated  against  each  of  these  three  cri¬ 
teria  on  a  Likert  scale  of  one  to  five;  "one"  Indicated  a  low  de¬ 
gree  of  Importance,  need,  or  likelihood  of  use,  while  a  "five"  In¬ 
dicated  a  high  degree  of  each  of  those  three  attributes. 

Each  respondent  was  then  asked  to  make  a  decision  as  to  the  Impor¬ 
tance  of  Individual  research  projects  listed  under  each  research 
area;  a  check  mark  Indicated  that  It  was  considered  an  Important 
project  to  undertake. 

Finally,  each  respondent  was  asked  to  open-ended  comments  about 
each  research  area  or  project  and  for  suggestions  of  potential 
funding  sources.  (See  Appendix  B  for  questionnaire  respondents.) 
The  subsequent  analysis  of  the  returned  questionnaires  is  found  In 
Appendix  C. 

4.  The  last  step  of  the  prioritization  process  involved  the  presenta¬ 
tion  of  the  questionnaire  results  of  the  Project  Guidance  Team  and 
the  facilitation  of  a  day-long  meeting.  During  this  time  the  PGT 
decided  upon  the  top  priority  research  areas  and  projects  and 
developed  plans  for  their  future  involvement  in  implementing  the 
research  plan. 


II.  FINDINGS  AND  RESULTS  OF  THE  PRIORITIZATION  PROCESS 


The  results  of  the  questionnaire  (see  Table  1,  2  and  3  on  the  following 
pages)  indicated  that  of  the  19  research  areas,  four  had  been  highly  ranked 
by  all  the  respondent  groups,  ten  research  areas  fell  into  the  mid-range 
rankings,  and  five  were  clearly  considered  to  be  of  less  immediate  impor¬ 
tance  in  terms  of  seeking  out  immediate  funding. 


Priority  Research  Areas 

The  four  top  priority  research  areas  were: 


Area  No.  5:  Provide  Adequate  Compensation  to  Enhance  Recruiting,  Re¬ 
tention,  and  Motivation 

Area  No.  12:  Develop  Effective  Managers  and  Supervisors 

Area  No.  9:  Develop  Effective  Systems  for  Classification  and 
Appraisal 


Area  No.  13:  Integrate  New  Technology  Within  Civilian  Workforce 

For  each  of  the  top  priority  areas,  a  feasibility  analysis  was  conducted. 
This  analysis  suggested  a  logical  sequencing  of  specific  research  projects 
to  be  carried  out  within  each  research  area,  estimates  of  the  time  and  cost 
parameters,  and  possible  funding  sources.  The  information  concerning  fund¬ 
ing  sources  was  derived  from  the  suggestions  of  the  questionnaire  respon¬ 
dents. 


TABLE  1 


K 

SUMMARY: 

PRIORITIZATION  FINDINGS, 

ALL  RESPONDENTS 

• 

- 

Research  Area 

Combined  Score 

"la 

portance* 

"Need" 

"Use" 

t 

1. 

No.  5 

(Compensation) 

3.57 

4.17 

3.64 

2.92 

2. 

No.  12 

(Effective  Managers) 

3.52 

4.28 

3.19 

3.08 

— 

3. 

No.  9 

(Classification) 

3.49 

3.78 

3.42 

3.28 

• 

4. 

No.  13 

(New  Technology) 

3.43 

3.75 

3.64 

2.89 

* 

5. 

No.  4 

(Selection  &  Screening) 

3.24 

3.69 

2.86 

3.17 

- 

6. 

Nos.  3  and  7 
(Models;  Marketing) 

3.20 

3.64 

3.39 

2.92 

3.28 

3.06 

2.94 

ft 

7. 

No.  1 

(Optimum  Mixes) 

3.16 

3.58 

3.44 

2.44 

• 

8. 

No.  15 

(Personnel  Practices) 

3.12 

3.50 

3.08 

2.86 

B 

9. 

No.  18 

( Retai n  Mi nori ti es) 

3.10 

3.39 

3.14 

2.42 

| 

10. 

No.  17 

(Recruit  Minorities) 

3.07 

3.31 

2.89 

3.03 

11. 

No.  19  3.04 

(Insure  Non-Discrimination) 

3.50 

2.69 

2.92 

» 

12. 

No.  10 
( Incentives) 

2.98 

3.39 

3.14 

2.42 

:  ■ 

13. 

No.  16 

(Productivity) 

2.93 

3.31 

3.00 

2.47 

» 

14. 

No.  11 

2.75 

3.00 

2.75 

2.50 

(Reduce  Barriers) 
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1 

NO.  5 

( Condensation) 

No.  9 

(Classifi  cation 

No.  13 

(New  Technology) 

No.  12 
(Managers) 

2 

No.  4,  No.  9 

(Selection; 

Classification) 

No.  5 

(Condensation) 

No.  1 

(Optimum  Mixes) 

No.  15 

(Personnel  Practices) 

3 

Nos.  17,  18 
(Recruit  and  Retain 
Minorities) 

No.  15 

(Personnel  Practices) 

No.  12 
(Managers) 

No.  4 

(Selection) 

4 

Nos.  3,  12 
(Models;  ffenagers) 

No.  12 
(Managers) 

No.  5 

(Condensation) 

No.  3 
(Madels) 

5 

No.  7 

(Marketing) 

No.  13 

(New  Technology) 

No.  19 

(Non-Discrimina¬ 

tion) 

No.  13 

(New  Technology) 

6 

No.  1 

(Optimum  Mixes) 

No.  19 

( Non-Di  scrinri  nation) 

No.  7 

(Marketing) 

No.  9;  No.  10 

(Classification; 

Incentives) 

7 

No.  13 

(New  Technology) 

No.  4 

(Selection) 

No.  9 

(Classification) 

No.  14 

(Increase  KSA's) 

8 

No.  15 

(Personnel  Practices) 

No.  18 

(Retain  Mi  non  ties) 

No.  10 
( Incentives) 

No.  5 

(Condensation) 

9 

No.  16 

(Productivity) 

No.  7 

(Marketing) 

No.  16;  No.  3 
Productivity, 

M)dels) 

No.  7 

(fferketing) 

10 

No.  19 

(Non-Oiscrinri nation) 

No.  17,  No.  6 
(Recruit;  Environment) 

No.  6 

(Job  Environment) 

NO.  16,  No.  17 
(Productivity 

Recruit) 

11 

No.  10 
( Incentives) 

No.  11 

(Reduce  Barriers) 

No.  18 

(Retain  Minorities) 

No.  1 

(Optimum  Mixes) 

RAKING  CF  RESEARCH  AREA  W  EACH  GROIP 

(Continued) 


Priority 

Qroup  1 

Qnaj>  2 

Qnoup  3 

Qroup4 

Rank 

(Policy  Personnel) 

(Personnel  Specialists) 

(Program  tanagers) 

(Other) 

12 

No.  8 

No.  3 

NO.  11 

No.  18 

(Career  Development) 

(Mxiels) 

(Reduce  Barriers) 

(Retain  Minorities) 

13 

No.  11 

No.  10,  No.  8 

No.  17 

No.  8 

(Reduce  Barriers) 

(Incentives;  Career) 

(Recruit  Minorities)  (Career  Development) 

14 

No.  14 

No.  16 

No.  4 

No.  11 

(Increase  KSA's) 

(Productivity) 

(Selection  and 
Screening) 

(Reduce  Barriers) 

15 

No.  2,  No.  6 

No.  14 

No.  14 

No.  19 

(Optinun  Ratios; 

Job  Environments) 

(Increase  KSA's) 

( Increase  KSA1 s) 

( Non-Di  scrimi  nati  on ) 

16 

No.  1 

No.  15 

No.  2 

(Optinun  Mixes) 

(Personnel 

Practices) 

(Optinun  Ratios) 

17 

No.  2 

No.  8 

No.  6 

(Optinun  Ratios) 

(Career  Development)  (Job  Envirorment) 

18 

No.  2 

(Optitrun  Ratios) 


TABLE  3 

¥EM  LIKERT  SCORES 


Priority 

Group  1 

Qnou)  2 

Qnoup  3 

4 

to* 

(Pollqy Personnel)  (Personnel  Specialists) 

(Other) 

■ 

1 

3.94 

4.05 

3.49 

4.08 

c. 

• 

(Gonpensation) 

(Classification) 

(New  Technology) 

(ftenagers) 

2 

3.64;  3.64 

3.81 

3.46 

4.00 

(Selection;  Classi- 

(Conpensation) 

(Optinun  Mixes) 

(Personnel  Practices) 

fication) 

m 

3 

3.47;  3.47 

3.67 

3.36 

3.92 

(Recruit,  Retain 

(Personnel  Practices) 

(Onagers) 

(Selection) 

Minorities) 

4 

3.44;  3.44 

3.62 

3.21 

3.83 

» 

(Mxiels;  ffenagers) 

(Itenagers) 

(Conpensation) 

(Mxiels) 

5 

3.38 

3.52 

3.13 

3.58 

(fferketing) 

(New  Technology) 

(Non-Discrimination) 

(New  Technology) 

a 

6 

3.33 

3.43 

3.08 

3.50;  3.50 

9 

(Optinun  Mixes) 

( Non-Di scrimi nati on) 

(terketing) 

(Gassi  fication; 
Incentives) 

7 

3.25 

3J8 

3.05 

3.33 

■ 

(New  Technology) 

(Selection) 

(Classification) 

( Increase  KSA' s) 

9 

8 

3.17 

3.24 

3.03 

3.25 

(Personnel  Practices) 

(Retain  Minorities) 

( Incentives) 

(Conpensation) 

9 

3.08 

.  3.19 

2.97;  2.97 

3.08 

i 

(Productivity) 

( targeting) 

(Prodictivity; 

Mxiels) 

(Marketing) 

• 

10 

3.00 

3.09;  3.09 

2.85 

3.0;  3.0 

( Non-Di  scrimi  nation) 

(Recruit;  Envirorment) 

(Job  Envi nonrents) 

(Productivity, 

Recruit) 

11 

2.97 

2.95 

2.82 

2.92 

• 

( Incentives) 

(Reduce  Barriers) 

(Retain  Minorities) 

(Optinun  Mixes) 

12 

2.91 

2.86 

2.79 

2.67 

(Career  Development) 

(Mxiels) 

(Reduce  Barriers) 

(Retain  Minorities) 

13 

2.75 

2.62;  2.62 

2.71 

2.33 

• 

(Reduce  Barriers) 

x  incentives;  Career) 

(Recruit  Minorities)  (Career  Development) 
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TABLE  3 

KAN  LIKERT  9 
(Continued) 


Priority 

Rank 

1155*1111 

Group  3 

(Propw  Managers) 

Group  4 
(Otter) 

14 

2.72 

(Increase  KSA's) 

2.52 

(Productivity) 

2.59 

(Selection  and 
Screening) 

2.25 

(Reduce  Barriers) 

15 

2.66;  2.66 
(Optimum  ratios; 
Job  environnents) 

2.48 

(Increase  ICSA's) 

2.54 

(Increase  KSA's) 

2.12 

( Non-Dl scrinri nation) 

16 

2.43 

(Optimum  Mixes) 

2.51 

(Personnel 

Practices) 

2.08 

(Optinun  Ratios) 

17 

2.14 

(Optinun  Ratios) 

2.49 

(Career  Development) 

2.00 

(Job  environment) 

18 

2.26 

(Optinun  Ratios) 
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AREA  NO.  5:  PROVIDE  ADEQUATE  COMPENSATION  TO  ENHANCE  RECRUITING,  RETEN¬ 
TION,  AND  MOTIVATION 


Promising  Projects 

1.  Identify  Compensation- related  Factors  Affecting  Recruitment,  Reten¬ 
tion,  and  Motivation 

a.  Review  literature  on  predictors  of  DoD  civilian  recruitment  and 
retention,  as  well  as  literature  on  measures  of  motivation  and 
extract  key  findings  on  compensation. 

b.  Design  and  conduct  surveys  of  targeted  samples  of  personnel  in 
key  occupations  of  varying  ages  and  grades  (also  could  include 
race  and  sex)  to  determine  perceived  importance  of  factors 
leading  to  decisions  to  enter,  stay  in,  and  leave  Navy  civilian 
service.  Concentrate  on  salary  issues  and  benefits  (current 
and  hypotheticals),  but  other  job  environment  factors  could  be 
Included. 

c.  Analyze  data,  and  identify  key  compensation- related  factors  af¬ 
fecting  recruitment  of  new  employees  and  those  affecting  reten¬ 
tion  and  motivation  of  current  employees. 

2.  Determine  Impacts  of  Employee  Turnover 

a.  Analyze  current  Navy  retention  rates  in  key  occupational  series 
by  grade  and  length  of  service. 


b.  Determine  turnover  costs,  based  on  replacement/ retraining  costs 
and  lost  productivity. 


3.  Compare  Federal  vs.  Private  Sector  Salaries  and  Benefits 


a.  Review  previous  methodologies,  identify  shortcomings,  and  re¬ 
commend  improved  approach. 

b.  Conduct  new  study  for  selected  occupations. 

4.  Develop  Models  to  Predict  Recruitment  and  Retention  Behavior  Based 

on  Compensation  Factors 

a.  Review  OSD  models  under  development  and  COPES  Project  and  ad¬ 
just  retention  models  as  necessary  based  on  findings  from  Pro¬ 
jects  1  and  3. 

b.  Develop  recruitment  model  based  on  data  from  Projects  1  and  3 
and  from  historical  personnel  records. 

5.  Identify  Feasible,  High-Payoff  Changes  in  Compensation  Structure 

a.  Review  outcomes  of  Projects  1  and  4  to  identify  changes  most 
likely  to  positively  impact  recruitment,  retention,  and  motiva¬ 
tion. 

b.  Analyze  the  cost-benefit  of  these  changes  using  Project  2  re¬ 
sults  and  Project  4  models. 

c.  Assess  feasibility  of  changes  and  identify  opportunities  for 
making  changes  or  designing  experiments  to  test  the  true  impact 
of  selected  changes. 


Approximate  Time  and  Cost  Parameters 

Project  1:  18-24  months 

$300  -  450,000 
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Note:  Costs  vary  according  to  number  and  size  of  survey  sam¬ 
ples.  Substitution  of  focus  groups  could  reduce  costs. 

Project  2:  6-12  months 
$75  -  200,000 

Project  3:  8-24  months 
$75  -  350,000 

Note:  Lower  ranges  assume  use  of  existing  data  sets  only. 

Project  4:  12-18  months 

$150-300,000 

Note:  Wait  for  completion  of  above  projects  and  OSD  study 

Project  5:  6-9  months 

$40  -  100,000 

Note:  Conducted  after  Projects  1,  2  and  4 

Funding  Sources:  0PM 
0P-01 

AREA  NO.  12:  DEVELOP  EFFECTIVE  MANAGERS  AND  SUPERVISORS 
Promising  Projects 

1.  Determine  Critical  Knowledge,  Skills,  and  Abilities  (KSA's) 

a.  Classify  managerial  positions  according  to: 

•  Level  (i.e.,  executives,  middle  managers,  front  line  super¬ 
visors);  and 


•  Work  Group  Type  (e.g.,  Lab,  Shipyard,  NARF,  HQ  Policy) 
and  select  categories  of  greatest  priority. 

b.  For  selected  categories,  review  literature  to  determine  best 
relevant  measures  of  managerial /supervisory  effectiveness  and 
potential  key  KSA's. 

c.  Validate  KSA's  for  Navy  civilian  population  through  records  re¬ 
view  and  interviewing. 

2.  Develop  Optimum  Selection  Criteria  and  Procedures 

a.  Identify  current  selection  criteria  and  procedures  and  assess 
relative  to  findings  from  Project  1. 

b.  Design  and  test  model  selection  criteria  and  procedures. 

3.  Develop  Optimum  Training/Development  Strategies 

a.  Identify  current  training  and  development  programs  and  assess 
relative  to  findings  from  Project  1  and  findings  from  litera¬ 
ture  on  management  training  effectiveness. 

b.  Identify  design  parameters  for  more  effective  Navy  training/ 
development  programs. 

c.  Develop,  test  and  evaluate  curricula  for  selected  categories  of 
personnel . 


ADDroximate  Time  and  Cost  Parameters 


Project  1:  12-18  months 

$175  -  400,000 


Project  2:  24  -  30  months 

$200  -  500,000 


Project  3:  24-30  months 

$200  -  600,000 

Note:  Costs  for  all  three  projects  can  vary  significantly  according 
to  the  number  of  categories  of  managers/supervisors  selected 
for  research  and  development  tasks.  Initial  phase  of  Projects 
2  and  3  could  be  conducted  simultaneously  with  the  latter 
phases  of  Project  1. 

Funding  Sources:  6.3 

0  and  MN 
0P-01 
MAT  01M 
Sy scorns 


AREA  NO.  9:  DEVELOP  EFFECTIVE  SYSTEMS  FOR  CLASSIFICATION  AND  APPRAISAL 
Promising  Projects: 

1.  Assess  Current  Systems 

a.  Review  recent  literature  linking  classification  and  appraisal 
systems  characteristics  to  employee  retention,  motivation  and 
productivity. 

b.  Establish  valid  measures  of  time  and  costs  required  to  adminis¬ 
ter  systems. 

c.  Assess  perceptions  of  current  systems  (including  BPAP,  Merit 
Pay  and  to  a  less  critical  extent  SES)  by  supervisors  and  em¬ 
ployees  in  selected  occupations,  including  perceived  impacts  of 


systems  on  motivation,  productivity  and  retention.  Identify 
any  opportunities  to  evaluate  behavioral  Impact  based  on  varia¬ 
tions  in  implementation. 


d.  Identify  strengths  and  weaknesses  of  current  systems  and  key 
characteristics  desired  in  optimum  system. 

2.  Evaluate  Replicability  of  Navy  Demonstration  Project 

a.  Review  of  evaluation  of  Demonstration  to  identify  unique  and 
potentially  replicable  site  characteristics  that  contribute  to 
success. 

b.  Identify  potential  new  sites,  assess  transferability  issues, 
and  develop  replication  strategy. 

c.  Design  and  conduct  experimental  replications  and  evaluate  im¬ 
pact  compared  to  original  and  control  sites,  and  develop  recom¬ 
mendations  for  future. 

3.  Assess  Alternative  Private  Sector  Systems 

a.  Review  private  sector  systems  and  identify  those  that  best  cap¬ 
ture  the  key  characteristics  identified  in  Project  1. 

b.  Compare  performance  measures  of  private  sector  systems  to 
established  measures  of  current  Federal  systems  and  the  Demon¬ 
stration  Project  (through  literature  review  and  selected  test¬ 
ing  as  necessary). 

c.  Identify  systems  with  greatest  performance  potential,  and  as¬ 
sess  opportunities  to  introduce  these  systems  or  to  test  them 
in  Navy  environment. 
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Approximate  Time  and  Cost  Parameters 


Project  1:  12-18  months 

$150  -  250,000 

Project  2:  36-48  months 
$300  -  500,000 

Note:  Cost  sensitive  to  number  of  replication  sites 

Project  3:  8-12  months 

$100  -  200,000 

Note:  Follows  completion  of  Project  1 

Funding  Sources:  6.2 
6.3 

0  and  MN  MPT 
0PNAV-.0P-01 
0PM 
NAVMAT 


AREA  NO.  13:  INTEGRATE  NEW  TECHNOLOGY  WITHIN  CIVILIAN  WORKFORCE 
Promising  Projects 

1.  Project  Impacts  of  Future  Technologies 

a.  Identify  major  technological  changes  that  are  likely  to  affect 
the  Navy  over  the  next  10  years,  particularly  including  small 
computer  systems  development  and  applications  and  robotics. 

b.  Assess  likely  impacts  of  these  technologies  on  current  proce¬ 
dures  for  accomplishing  tasks  in  the  specific  work  areas  most 
affected. 


c.  Identify  the  principal  new  knowledge,  skills  and  abilities  that 
will  be  required  and  determine  the  major  implications  for  re¬ 
cruitment,  training,  and  re-training. 

d.  Review  literature  and  assess  attitudes  on  organizational  accep¬ 
tance/adaptation  to  new  technology;  identify  key  principles  to 
guide  future  implementation  within  the  Navy  in  areas  identi¬ 
fied. 

2.  Model  Program  Testing 

a.  Design  a  model  program,  involving  employee  education,  training, 
and  organizational  adjustment,  for  the  introduction  of  a  speci¬ 
fic  technology  in  a  specific  work  area. 

b.  Test  and  evaluate  the  model  and  assess  replicability. 

3.  Assess  Opportunities  for  Computer-Aided  Training  (lower  priority) 


Approximate  Time  and  Cost  Parameters 

Project  1;  12-18  months 

$150  -  500,000 

Note:  This  project  could  be  done  broadly  across  the  Navy  as 
written,  or  for  selected  work  areas. 

Project  2:  15-24  months 

$150  -  300,000 

Potential  Funding  Sources:  6.3 

0  and  MN 
SYSCOMS 


MID-RANGE  RESEARCH  AREAS 


The  mid-range  ranked  research  areas,  with  those  specific  research  projects 
chosen  by  at  least  1/3  of  the  respondents,  were  then  compiled,  along  with 
funding  source  suggestions,  as  shown  below: 

Area  No.  4:  Develop  Effective  Selection  and  Screening  Mecha¬ 
nisms 

Develop  measures  of  quality/ performance  po¬ 
tential 

Identify  attributes  associated  with  high  per¬ 
formance 

Identify,  develop,  and  test  effective  tools 
Funding:  6.2 
6.3 
OP.  01 


Area  No.  3:  Develop  Credible  Models  for  Predicting  Civilian 
Personnel  Needed  for  actual  and  contingency  Navy 
settl ngs 

Review  and  critique  existing  forecasting 
model s 

Develop  most  credible  model 
Develop  strategies  to  integrate  forecasting 
into  budget 
Funding:  6.2 
6.3 

MPT/Logistics 

OP-1 


Area  No .  7 : 


Use  Effective  Marketing  Strategies  for  Recruitment 
Assess  changes  In  labor  pools  in  future 
Identify  promising  sources  of  future  recruits 
for  selected  occupations 
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Develop  and  Test  effective  outreach  strate 
gies 

Funding:  0P-01 
0PM 

Area  No.  1:  Develop  Optimum  Mixes  of  Military,  Civilian,  and 
Contracted  Personnel 

Determine  current  nature  and  extent  of  con¬ 
tracting  out 

Determine  cost  effectiveness  of  contracting 
out 

Determine  optimum  mixes  of  military  and  ci¬ 
vilian  personnel 
Funding:  6.3 

MPT/Logistics 

OP-01 

NAVCOMP 

Area  No.  15:  Increase  Efficiency  of  Selected  Personnel  Practices 

Measure  Time  and  Costs  Associated  with  ULP 
grievances 

Measure  Time  and  Costs  Associated  with  Other 
Personnel  Functions 
Funding:  0P-01 

0  and  MN 

Area  No.  18:  Retain  Adequate  Numbers  of  Quality  Minority  Person¬ 
nel 

Identify  reasons  why  minorities  stay  or  leave 
Funding:  0  and  MN 
0P-01 
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Area  No.  17:  Recruit  Adequate  Numbers  of  Quality  Minority  Per¬ 
sonnel 

Identify  potential  institutional  barriers, 
e.g.,  community,  veterans  preference, 
exams,  physical  barriers,  contracting  out 
Identify  current  successful  recruiting  stra¬ 
tegies  and  assess  transferability 
Evaluate  cost-effective  of  EEO  structure  in 
meeting  EEO  objectives 
Funding:  0  and  MN 
0P-01 

Area  No.  19:  Insure  Non-Discrlminatory  Treatment  of  Minorities 

Identify  current  levels,  trends,  and  patterns 
of  discrimination  complaints 
Identify  and  test  promising  procedures  or 
training  programs  to  reduce  discrimination 
compl ai nts 

Evaluate  the  cost-effectiveness  of  the  EEO 
structure  in  meeting  this  objective 
Funding:  None  listed 

Area  No.  10:  Develop  Individual  Incentives  to  Increase  Motiva¬ 
tion 

Develop  valid  measures  of  motivation/morale 
and  productivity  and  determine  relationship. 
Identify  and  evaluate  current  Navy  techniques 
Identify  successful  private  sector  techniques 
Identify  potential  new  techniques  and  assess 
feasibility 

Area  No.  16:  Develop  Organizational  Structures  That  Enhance  Pro¬ 
ductivity 

Identify  differences  between  military  and 
civilian  leadership  styles  and  impact  on 
productivity 


.  • 
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Funding:  OP-01 
0P-04 
MAT  01M 


LOWEST  RANKED  RESEARCH  AREAS 

Finally,  those  research  areas  ranked  as  least  important  by  the  respondents 
were: 

Area  No.  11:  Reduce  Barriers  to  Individual  Motivation 
Funding:  NAVMAT  R&D 

Area  No.  6:  Provide  Satisfying  Job  Environments  to  Improve  Job 
Acceptance  Rates,  Turnover,  Motivation/Productivity 
Funding:  0  and  MN 

Area  No.  14:  Increase  Knowledge,  Skills,  and  Abilities  of  Civil¬ 
ian  Personnel 

Funding:  0  and  MN;  0P-01;  NAVMAT 

Area  No.  8:  Provide  Satisifying  Opportunities  for  Career  De¬ 
velopment 

Funding:  6.2  and  6.3,  0  and  MN 

Area  No.  2:  Develop  Optimum  Ratios  of  Supervisors  to  Employees 
Funding:  0P-01;  MAT  01M 

Project  Guidance  Team  Decisions 

The  final  stage  of  the  prioritization  project  began  with  a  presentation  of 
the  results  of  the  mail  questionnaire  to  the  Project  Guidance  Team.  In  a 
day-long  discussion  session,  facilitated  by  the  contractor,  the  group  re¬ 
viewed  the  results  of  the  questionnaire,  and  then  reached  consensus  in 
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terms  of  research  areas  and  projects  for  which  resources  would  Initially  be 
sought.  The  top  priority  areas  chosen  by  the  PGT  were: 

e  Develop  Effective  Managers  and  Supervisors 

•  Provide  Adequate  Compensation  to  Enhance  Recruiting,  Reten¬ 
tion,  and  Motivation 

•  Develop  Optimum  Mixes  of  Military,  Civilian  and  Contracted 
Personnel 

e  Develop  Effective  Selection  and  Screening  Mechanisms 
e  Develop  Effective  Systems  for  Appraisal 
e  Develop  Individual  Incentives  to  Increase  Motivation 

•  Integrate  New  Technology  into  Civilian  Workforce 

•  Recruitment  and  Retain  Adequate  Numbers  of  Quality  Minority 
Personnel . 

The  PGT  also  decided  to  form  workgroups  for  each  research  area.  The  task 
of  these  workgroups  would  be  to  develop  more  closely  defined  and  targeted 
research  projects. 
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APPENDIX  A 
QUESTIONNAIRE 


PRIORITIZATION  INSTRUCTIONS 


The  questionnaire  on  the  following  pages  Is  made  up  of  19  research  areas 
contained  In  the  Roadmap.  Under  each  general  area  are  specific  research 
topics  suggested  by  the  study  participants.  The  numbers  In  parentheses 
following  each  research  topic  Indicate  the  pages  In  the  Roadmap  where  each 
research  topic  Is  discussed  In  more  detail. 

Please  read  each  research  area  first  and  rank  It  on  the  basis  of  three  cri¬ 
teria. 

The  first  criterion  is  the  relative  importance  to  the  Navy  of  improving  ef¬ 
fectiveness  in  a  particular  area  on  a  1  to  5  scale.  A  low  score  indicates 
that  improving  current  effectiveness  will  make  little  contribution  to  the 
Navy's  mission,  while  a  high  score  Indicates  that  increased  effectiveness 
would  contribute  greatly  to  the  Navy's  mission. 


DEVELOP  TIMELY  PERSONNEL  PROCEDURES. 


1 

2 

3 

4 

5 

Slightly 

Moderately 

Very 

Extremely 

Important 

Important 

Important 

Important 

Important 

The  second  criterion  is  the  relative  need  for  new  information  in  order  to 
improve  capabilities  in  a  particular  area.  A  low  score  indicates  that  the 
information  needed  to  improve  effectiveness  is  already  available;  a  high 
score  indicates  that  new  information  or  understanding  is  necessary  before 
effective  action  can  be  taken. 


DEVELOP  TIMELY  PERSONNEL  PROCEDURES. 

1 

2  3 

4 

5 

Little  Need 

Some  Need  Needed 

Important  Need 

Critical  Need 

A-l 


The  last  criterion  Is  your  perception  of  the  1 Ikel Ihood  that  research  find¬ 
ings  will  be  util  1 zed  effectively  once  they  are  obtained.  A  low  score  in¬ 
dicates  the  respondent's  belief  that  the  political,  financial,  or  organiza¬ 
tional  climate  is  such  that  here  is  little  chance  that  change  will  be 
made.  A  high  score  indicates  that  the  area  Is  "hot",  and  there  Is  excel¬ 
lent  potential  for  action  based  on  research  or  study  findings. 

DEVELOP  TIMELY  PERSONNEL  PROCEDURES. 

1  2  3  4  5 

Not  likely  Somewhat  Likely  Likely  Very  Likely  Extremely  Likely 

To  the  right  of  each  research  area  are  3  boxes.  In  the  A  box,  indicate 
your  scoring  of  “importance";  the  B  box  is  for  your  scoring  of  “need  for 
new  information";  the  C  box  is  for  "likelihood  of  use".  For  example: 


DEVELOP  TIMELY  PERSONNEL  PROCEDURES. 


This  would  indicate  that  the  respondent  thinks  that  it  is  very  important  to 
improve  effectiveness  in  the  area  of  more  timely  personnel  procedures,  that 
only  some  new  information  is  needed  in  order  to  improve  effectiveness,  but 
it  is  not  likely  that  research  findings  would  actually  be  utilized  to  de¬ 
velop  such  procedures. 

The  specific  research  topics  listed  under  each  general  area  have  a  space 
provided  to  the  left.  Please  check  those  research  topics  which,  in  your 
opinion,  are  the  most  important  ones  to  study.  For  example: 


Determine  current  time  required  for  filling  vacant  positions 
for  varous  occuptions  and  activities. 


_  Identify  techniques  utilized  by  Navy  activities  that  fill  va¬ 
cancies  most  quickly,  and  assess  transferability. 

_  Examine  grievance  handling  procedures  developed  in  the  private 

sector  and  assess  transferability. 

Check  as  many  or  as  few  of  these  as  you  wish,  and  include  comments  or  sug¬ 
gestions  if  desired. 

In  the  line  following  the  "comments"  section,  please  indicate  any  potential 
funding  source  for  the  research  area. 

We  very  much  appreciate  your  cooperation  in  this  project.  The  question¬ 
naire  should  not  take  more  than  a  half  hour  to  complete.  Please  return  it 
within  a  week.  If  you  respond  to  it  in  collaboration  with  other  members  of 
your  staff,  please  indicate  their  names  and  position  titles  on  the  last 
page  of  the  questionnaire. 

Thank  you! 
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A  *  Importance;  B  =  Need  For  New  Information;  C  *  Like! i hood  of  Use 
Rate  from  1  (very  low)  to  5  (very  htgh). 

’  ABC 

1.  DEVELOP  OPTIMUM  MIXES  OF  MILITARY,  CIVILIAN,  AND  CONTRACTED  PERSONNEL. 

la.  _  Determine  current  nature  and  extent  of  contracting  out.  (20) 

lb.  _  Determine  cost  effectiveness  of  contracting  out.  (20) 

lc.  _  Determine  current  mixes  of  military  vs.  civilian  personnel.  (21) 

ld.  _  Determine  optimum  mixes  of  military  vs.  civilian  personnel.  (21) 

Comments : 


Funding  Source: _ 

ABC 

2.  DEVELOP  OPTIMUM  RATIOS  OF  SUPERVISORS  TO  EMPLOYEES. 

2a.  _  Assess  current  ratios,  including  civilian  supervision  of  contractors.  (26) 

2b.  _  Develop  standards  for  supervisory  ratios.  (26) 

Comments: 


Funding  Source: _ 


ABC 

3.  DEVELOP  CREDIBLE,  USABLE  MODELS  FOR  PREDICTING  CIVILIAN  PERSONNEL  - - . — 

NEEDED  FOR  ACTUAL  AND  CONTINGENCY  NAVY  SITUATIONS. 


3a.  _  Review  and  critique  existing  civilian  and  military  forecasting  models.  (26) 

3b.  _  Develop  most  credible  model.  (26) 
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A  *  Importance;  B  *  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 


DEVELOP  CREDIBLE,  USABLE  MODELS  FOR  PREDICTING  CIVILIAN  PERSONNEL 
NEEDED  FOR  ACTUAL  AND  CONTINGENCY  NAVY  SITUATIONS.  (CONTINUED) 


3c.  Document  current  processes  for  projecting  needs  and  formulating 

budgets.  (27-28) 

3d.  Develop  strategies  to  integrate  forecasting  into  the  budget  development 

process.  (27-28) 

Comments : 


Funding  Source: 


SI 

B 


ABC 

4.  DEVELOP  EFFECTIVE  SELECTION  AND  SCREENING  MECHANISMS. 

4a.  _  Develop  measures  of  quality/performance  potential.  (31-32) 

4b.  _  Assess  current  process  and  current  success  rates  for  key  occupations.  (32) 

4c.  _  Identify  attributes  associated  with  high  performance.  (33) 

4d.  _  Identify,  develop,  and  test  effective  tools.  (38-39) 

Comments: 


Funding  Source: 


ABC 


5.  PROVIDE  ADEQUATE  COMPENSATION  TO  ENHANCE  RECRUITING,  RETENTION,  AND 
MOTIVATION. 


5a.  _  Determine  importance  of  different  benefits  by  age  and  grade.  (49) 

1  5b.  _  Determine  current  retention  success  rates  in  key  occupations.  (46,  48) 
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A  *  Importance;  B  *  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 


5.  PROVIDE  ADEQUATE  COMPENSATION  TO  ENHANCE  RECRUITING,  RETENTION,  AND 
MOTIVATION.  (CONTINUED) 


5c. 

5d. 

5e. 

5f. 

5g. 

5h. 

51. 

5j. 


5k. 


_  Determine  Importance  of  different  benefits  by  age  and  grade.  (49) 

_  Determine  current  retention  success  rates  in  key  occupations.  (46,  48) 

_  Determine  replacement  costs.  (48) 

_  Analyze  Impact  of  salary  and  benefits  on  job  acceptance  or  refusals.  (49-50). 

_  Analyse  Impact  of  salary  and  benefits  on  decision  to  leave  or  stay.  (49-50) 

_  Analyse  Impact  of  salary  and  benefits  on  motivation  and  productivity. 

_  Compare  Federal  vs.  private  sector  salary  and  benefit  levels.  (50) 

Identify  and  determine  feasibility  of  selected  changes  in  salary  and  benefits 
structure.  (56) 

_  Test  selected  changes.  (60) 

Comments: 


Funding  Source: 


ABC 

6.  PROVIDE  SATISFYING  JOB  ENVIRONMENTS  TO  IMPROVE  JOB  ACCEPTANCE  RATES,  - - — 

TURNOVER,  AND  MOTIVATION/PRODUCTIVITY. 


6a.  Determine  importance  of  job  content,  security,  and  advancement 

potential.  (49) 

6b.  _  Determine  importance  of  image  of  public  servant.  (49,  71) 

6c.  _  Determine  importance  of  physical  surroundings.  (50,  71) 

6d.  _  Determine  importance  of  flexible  hours.  (71) 

6e.  Determine  importance  of  perquisites  for  upper  level  employees.  (61) 


A  =  Importance;  B  *  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 


6.  PROVIDE  SATISFYING  JOB  ENVIRONMENTS  TO  IMPROVE  JOB  ACCEPTANCE  RATES, 
TURNOVER,  AND  MOTIVATION/PRODUCTIVITY.  (CONTINUED) 

6f.  _  Compare  Federal  vs.  private  sector  job  environment  features.  (50) 

Comments : 


Funding  Source: 


ABC 

7.  USE  EFFECTIVE  MARKETING  STRATEGIES  FOR  RECRUITMENT.  ' 

7a.  _  Assess  changes  in  labor  pools  in  future.  (37) 

7b.  _  Identify  current  sources  of  recruits  to  key  occupations.  (37) 

7c.  _  Identify  current  networks  for  finding  Federal  jobs.  (37) 

7d.  _  Identify  promising  sources  of  future  recruits  for  selected  occupations.  (37) 

7e.  _  Develop  and  test  effective  outreach  strategies.  (41) 

7f.  _  Identify  and  test  effective  uses  of  media  for  target  groups.  (42) 

Comments : 


P  Funding  Source: 


ABC 

8.  PROVIDE  SATISFYING  OPPORTUNITIES  FOR  CAREER  DEVELOPMENT.  I  1  T~ 


8a.  Identify  current  career  advancement  patterns.  (54-55) 

L 

t_  8b.  _  Identify  limits  to  advancement  at  upper  grades  and  for  senior 

technicians.  (54) 
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A  =  Importance;  B  =  Need  For  New  Information;  C  *  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 

8.  PROVIDE  SATISFYING  OPPORTUNITIES  FOR  CAREER  DEVELOPMENT.  (CONTINUED) 

8c.  Compare  Navy,  Navy  civilian,  and  private  sector  career  development 

patterns.  (58) 

8d.  _  Determine  impact  of  career  advancement  on  retention.  (54) 

8e.  _  Identify  needed  career  management  information.  (55) 

8f.  _  Design  central,  automated  system  to  match  personnel  with  job  openings.  (58) 

8g.  _  Evaluate  impact  of  special  training  and  education  on  career  advancement  and 

retention.  (60) 

Comments : 


Funding  Source: _ 

ABC 

9.  DEVELOP  EFFECTIVE  SYSTEMS  FOR  CLASSIFICATION  AND  APPRAISAL. 

9a.  _  Measure  time  and  costs  required  for  current  systems.  (53) 

9b.  _  Develop  position  management  strategies  to  increase  challenge  of  jobs.  (58) 

9c.  _  Compare  Navy  Demo  Project  to  alternative  private  sector  systems.  (60) 

9d.  _  Evaluate  replicability  of  Navy  Demo  Project  at  other  activities.  (60,  92) 

9e.  _  Evaluate  impact  of  SES  bonus  system  on  retention  and  productivity.  (58,  61) 

9f.  _  Evaluate  impact  of  merit  pay  on  retention  and  productivity.  (61,  71,  89) 

9g.  _  Evaluate  impact  of  Basic  Personnel  Appraisal  Program  on  retention  and 

productivity.  (61) 

Comments: 


Funding  Source: 
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A  =  Importance;  B  =  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 


ABC 


10.  DEVELOP  INDIVIDUAL  INCENTIVES  TO  INCREASE  MOTIVATION. 


I 


10a. 

10b. 

10c. 

lOd. 

lOe. 


Develop  valid  measures  of  motivation/morale  and  productivity  and  determine 
relationship.  (65,  66,  70) 

_  Identify  factors  that  affect  motivation.  (70,  71) 

_  Identify  and  evaluate  current  Navy  techniques  (including  monetary  incentives, 

recognition.  Navy  Demo  Project,  and  others).  (71,  89) 

_  Identify  successful  private  sector  techniques.  (83) 

_  Identify  potential  new  techniques  and  assess  feasibility.  (83) 

Comments: 


Funding  Source: 


ABC 


I 


11.  REDUCE  BARRIERS  TO  INDIVIDUAL  MOTIVATION. 


11a. 


Determine  impact  of  handling  hazardous  materials  on  productivity  and  develop 
and  test  educational  strategies  to  overcome.  (71) 


lib. 


> 


Determine  impact  of  drug  and  alcohol  problems,  evaluate  employee  assistance 
programs,  and  assess  opportunities  to  combine  with  military  programs.  (71, 
89) 


11c.  _  Determine  impact  of  consistent  low  performance  on  morale,  develop  procedures 

for  handling  low  performance  individuals.  (83) 


Comments: 


Funding  Source: 
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A  =  Importance;  B  =  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 

_  ~  ~  ABC 

12.  DEVELOP  EFFECTIVE  MANAGERS  AND  SUPERVISORS. 

12a.  _  Identify  critical  skills  and  abilities  needed.  (71,  75) 

12b.  _  Develop  measures  of  effectiveness.  (76,  86) 

12c.  _  Develop  discriminating  selection  criteria/procedures.  (76,  86) 

12d.  _  Determine  optimum  training/development  strategies.  (86) 

Comments: 


Funding  Source: 


ABC 

13.  INTEGRATE  NEW  TECHNOLOGY  WITHIN  CIVILIAN  WORKFORCE. 

13a.  Identify  impact  of  future  technology  on  future  requirements  for  skilled 

personnel.  (76) 

13b.  _  Identify  knowledge,  skills,  and  abilities  needed  to  use  new 

technologies.  (76) 

13c.  _  Develop  and  test  training  strategies  to  use  new  technologies.  (76) 

13d.  _  Assess  impacts  on  management  span  of  control.  (76) 

13e.  _  Assess  opportunities  for  Computer  Assisted  Training  and  test 

effectiveness.  (86,  92) 

Comments: 


Funding  Source: 


A  =  Importance;  B  =  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 


ABC 

14.  INCREASE  KNOWLEDGE,  SKILLS,  AND  ABILITIES  (KSA'S)  OF  CIVILIAN  - - — 

PERSONNEL. 


14a.  _  Develop  task  analyses  for  white  collar  jobs.  (75) 

14b.  Determine  return  on  investment  and  identify  techniques  for  mid-career 

training.  (92) 

14c.  _  Determine  impact  of  training  on  productivity/performance.  (92) 

Comments : 


Funding  Source: 


ABC 

15.  INCREASE  THE  EFFICIENCY  OF  SELECTED  PERSONNEL  PRACTICES.  I 


15a.  _  Measure  time  and  costs  associated  with  unfair  labor  practices/ 

grievances.  (87) 

15b.  Measure  time  and  costs  associated  with  other  personnel  functions.  (78,  79, 

80) 


15c.  Identify  and  test/evaluate  promising  techniques  for  handling  ULP 

grievances.  (87) 

15d.  _  Identify  needs  for  NCPDS  data  and  develop  plan  for  its  effective  use.  (88) 

Comments: 


Funding  Source: 
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A  =  Importance;  B  =  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  {very  low)  to  5  (very  high). 


K 


A 


B 


C 


16.  DEVELOP  ORGANIZATIONAL  STRUCTURES  THAT  ENHANCE  PRODUCTIVITY. 

16a.  Identify  differences  between  military  and  civilian  leadership  styles  and 

impact  on  productivity.  (80) 

16b.  Identify  organizational  and  training  options  to  reduce  military-civilian 

dissonance.  (87) 

16c.  _  Determine  time  and  costs  of  downtime  in  ship  maintenance.  (80,  92) 

16d.  Identify,  test,  and  evaluate  cross-training  and  work  team  options  to  reduce 

downtime.  (92) 

Comments: 


Funding  Source: 


17. 


ABC 

RECRUIT  ADEQUATE  NUMBERS  OF  QUALITY  MINORITY  PERSONNEL. 


i 

17a. 

17b. 

• 

17c. 

17d. 

17e. 

0 

r  ' 

17  f . 

f 

t  ' 

17g. 

Determine  why  current  minorities  chose  Navy  career.  (101,  102) 

Identify  potential  institutional  barriers  to  recruitment,  e.g.,  community  and 
educational  barriers,  veterans  preference,  exams,  physical  barriers  for  the 
handicapped,  contracting-out.  (103) 

Identify  current  successful  Navy  recruiting  strategies  and  assess 
transferabil i ty.  (105) 

Identify  promising  sources  of  recruits.  (105) 

Identify  changes  in  work  environment  to  increase  opportunities  for  physically 
disabled.  (107) 

Evaluate  cost-effectiveness  of  EEO  structure  in  meeting  EEO  objectives.  (109) 

Analyze  impact  of  minority  recruiting  on  productivity  and  identify  models  to 
relate  recruitment  to  priority  manpower  needs.  (104,  105) 
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A  =  Importance;  B  =  Need  For  New  Information;  C  =  Likelihood  of  Use 
Rate  from  1  (very  low)  to  5  (very  high). 


17.  RECRUIT  ADEQUATE  NUMBERS  OF  QUALITY  MINORITY  PERSONNEL.  (CONTINUED) 


17h. 


17  i . 


Test  long-run  cost-benefits  of  concentrating  recruiting  resources  on  high 
quality  recruits.  (109) 


Test  and  evaluate  promising  recruiting  strategies.  (105) 


Comments: 


Funding  Source: 


ABC 


18.  RETAIN  ADEQUATE  NUMBERS  OF  QUALITY  MINORITY  PERSONNEL. 


18a. 

18b. 

18c. 

18d. 

18e. 

18f. 


Develop  data  base  and  profiles  on  minorities  in  Navy  workplace.  (98,  100) 

Identify  reasons  why  minorities  stay  or  leave.  (102-103) 

Identify  minority  career  patterns  and  advancement  factors.  (108) 

Assess  minority  information,  needs  about  career  development,  and  test  model 
dissemination  approaches.  (108) 

Identify  and  examine  feasibility,  and  test  new  exams  or  procedures  to  enable 
minorities  to  enter  competitive  and  management  positions.  (108) 


Evaluate  cost-effectiveness  of  EEO  structure  in  meeting  this  objective.  (109) 


Comments: 


Funding  Source: 
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19a.  Identify  current  levels,  trends,  and  patterns  of  discrimination  complaints. 

(98) 


19b.  Identify  and  test  promising  procedures  or  training  programs  to  reduce 

discrimination  complaints.  (110) 

19c.  Evaluate  the  cost-effectiveness  of  the  EEO  structure  in  meeting  this 

objective.  (109) 

Comments: 


Funding  Source: 


NAME  AND  POSITION  OF  QUESTIONNAIRE  RESPONDENTS: 


APPENDIX  B 
RESPONDENTS 


APPENDIX  B 


QUESTIONNAIRE  RESPONDENTS 
ROADMAP  RESEARCH  PRIORITIZATION  PROJECT 


Mr.  Bun  B.  Bray,  Jr. 

Executive  Director 
Federal  Managers  Association 

Mr.  Ronald  C.  Burow 

Head,  Personnel  Management  and  Evaluation  Branch 
Civilian  Personnel  Policy  Division 

Vice  Admiral  James  B.  Busey 
Commander,  Naval  Air  Systems  Command 
Naval  Air  Systems  Command 

Mr.  Emerson  Cale 

Deputy  Director,  Logistics  Plans  and  Programs 
0P-40B 

Dr.  James  E.  Colvard 
Deputy  Chief  of  Naval  Material 
Navy  Material  Command 

Mr.  Tom  Cooper 
Director,  CCPO 
Charleston,  SC 

Dr.  Helen  Cristrup 
Acting  Assistant  Director 
Office  of  Staff  and  Policy 
Office  of  Personnel  Management 

Mr.  John  R.  Curran,  Sr. 

Director,  Civilian  Personnel 
Headquarters  Naval  Material  Command 

Dr.  Angelo  J.  D1  Mascio 
Deputy  Commander,  Naval  Air  Systems 
Command 

Naval  Air  Systems  Command 
Mr.  H.  Lee  Dixson 

Director,  Civilian  Manpower  Division 
Comptroller  of  the  Navy 
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QUESTIONNAIRE  RESPONDENTS 


(Continued)  < 


Mr.  James  P.  Early 
Director,  Civilian  Personnel 
Norfolk  Naval  Shipyard 

Mr.  Reginald  M.  Felton 

Head,  Equal  Employment  Opportunity  Programs  Branch 
Civilian  Personnel  Policy  Division 

Mr.  Terry  J.  Haycock 

Head,  Labor  and  Employees  Relations  Branch 
Civilian  Personnel  Policy  Division 

Rear  Admiral  A.  J.  Herberger 
Director,  Military  Personnel  Policy 
Division 

Office  of  the  Chief  of  Naval  Operations 

Mr.  Robert  M.  Hi 1 Iyer 
Director,  Navy  Laboratories 
Headquarters,  Naval  Material  Command 
(now  Technical  Director,  Naval  Ocean 
Systems  Center,  San  Diego) 

Mr,  A.  W.  Hines 

Associate  Director  of  Navy  Labs 

Mr.  Allen  Johnson 
Civilian  Personnel  Director 
Subase,  Banyor 

Mr.  Leonard  Klein 

Assistant  Deputy  Chief  of  Navy  Material 
(Manpower  and  Personnel) 

Naval  Material  Command 

Mr.  Conrad  Lacy,  Chief 
Management  Analysis  and 
Development  Office 

U.S.  Army  Civilian  Personnel  Center  (PECC-AD) 

Mr.  Larry  Lacy 

Deputy  Assistant  Secretary  of 
Defense  for  Civilian  Personnel 
Policy  and  Requirements 

Office  of  the  Assistant  Secretary  of  Defense  (MRA&L) 


QUESTIONNAIRE  RESPONDENTS 


(Continued) 


Mr.  Robert  Lehto 

Director,  Total  Force  Information 
Systems  Management  Division  (OP-16) 

Office  of  the  Chief  of  Naval  Operations 

Mr.  Thomas  E.  Lindsey 

Head,  Training  and  Career  Management  Branch 
Civilian  Personnel  Policy  Division 

Ms.  Sue  M.  Martin 

Head,  Executive  Personnel  and  Performance 
Appraisal  Systems  Branch 
Civilian  Personnel  Policy  Division 

Mr.  Kenneth  H.  McDaniel 
EEO  Manager 

Naval  Underwater  Systems  Center 
Newport,  RI 

Ms.  Dorothy  M.  Meletzke 

Director,  Navy  Civilian  Personnel  Command 

Captain  Phillip  Monroe 
Commanding  Officer 
Naval  Air  Rework  Facility 
Naval  Air  Station,  San  Diego 

Mr.  Marvin  Moss 
Technical  Director 
Office  of  Naval  Research 

Mr.  Thomas  R.  Muir 

Assistant  Deputy  Chief  of  Naval  Operations 
(Civilian  Personnel /Equal  Employment 
Opportunity) 


Dr.  Richard  J.  Niehaus 

Assistant  for  Human  Resources  Analysis 

Office  of  the  Chief  of  Naval  Operations 

Mr.  E.  W.  O'Brien 

Head,  Manpower  Mix  Branch,  OP-124 


QUESTIONNAIRE  RESPONDENTS 


(Continued) 


Ms.  Lorraine  G.  Ratto 
Assistant  for  Civilian  Personnel  Research 
Navy  Personnel  Research  and  Development 
Center 

Mr.  Edward  G.  Sherrill 

Director  of  Civilian  Personnel  Programs 

HDQRS,  United  States  Marine  Corps 

Mr.  George  P.  Stelnhauer 

Head,  Staffing  and  Pay  Systems  Branch 

Civilian  Personnel  Policy  Division 

Mr.  Joseph  K.  Taussig,  Jr. 

Deputy  Assistant  Secretary  of  the  Navy 
(Civilian  Personnel  Policy/Equal 
Employment  Opportunity) 

Dr.  James  W.  Tweeddale 
Technical  Director 

Navy  Personnel  Research  and  Development 
Center 

Mr.  Alfred  D.  White 
Special  Assistant,  EEO 
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APPENDIX  C:  METHODOLOGY 


Analysis  Plan 

The  analysis  of  the  data  resulting  from  the  questionnaire  consisted  of  two 
stages.  The  first  was  the  statistical  analysis  of  the  responses  while  the 
second  consisted  of  a  feasibility  analysis  of  each  of  the  highest  ranking 
areas  and  projects  in  terms  of  research  sequencing  and  estimated  time  and 
cost  parameters. 


STAGE  ONE:  STATISTICAL  ANALYSIS 

The  principal  objective  of  this  stage  of  the  analysis  was  to  identify  the 
group  of  research  areas  and  research  topics  within  research  area  ranked  the 
highest  by  the  respondents  to  the  survey  questionnaire.  Moreover,  our 
analysis  was  also  designed  to  test  the  hypothesis  that  the  rankings  by 
respondent  groups  are  independent  against  the  alternative  hypothesis  that 
there  is  concurrence  by  the  respondents  in  their  rankings.  The  statistical 
methods  used  to  analyze  the  research  area  and  research  topic  responses  are 
basically  the  same  and  are  discussed  below. 

Research  Area  Analysis:  Completed  surveys  for  the  analysis  were  received 
from  36  respondents.  Each  respondent  rated  each  of  19  research  areas  with 
a  Likert  score  ranging  from  one  (for  the  lowest  rating)  to  five  (for  the 
highest  rating).  Initially  we  created  a  36  by  19  data  array  containing  all 
of  the  Likert  score  responses  on  the  survey.  There  are  three  such  data 
surveys  since  each  research  area  was  rated  according  to  three  separate  cri¬ 
teria.  We  created  a  fourth  data  array  that  simply  represents  a  linear  com¬ 
bination  of  the  Likert  scores  given  to  each  research  area  by  each  respon¬ 
dent  sunned  across  each  of  the  three  objectives. 


The  column  sums  In  each  data  array  are  the  combined  Likert  ratings  for  a 
given  research  area  across  each  of  the  36  respondents  In  the  sample.  By 
dividing  these  column  sums  by  36  (108  by  the  data  array  representing  Likert 
scores  summed  over  each  of  the  three  objectives),  we  calculated  the  mean 
Likert  score  for  each  research  area.  The  mean  Likert  ratings  for  the  com¬ 
bined  responses  across  the  three  objectives  give  equal  weight  to  each  cri¬ 
terion.  We  used  these  mean  values  (or,  equivalently,  the  column  sums)  from 
the  combined-criteria  data  array  to  establish  the  final  set  of  rankings  for 
the  19  research  areas.  These  results  are  reported  in  table  one  in  the  next 
chapter.  In  addition  to  the  mean  Likert  score  across  all  three  research 
areas  and  the  associated  rankings,  the  mean  Likert  score  for  each  separate 
criterion  is  also  reported. 

To  test  whether  or  not  the  research  areas  can  be  distinguished  on  the  basis 
of  the  rankings,  we  first  combined  the  respondents  into  four  separate 
groups  according  to  their  general  job  classification.  The  Likert  scores 
were  then  summed  across  the  respondents  in  each  group  for  each  research 
area  and  for  each  criterion  on  which  the  research  areas  were  rated.  In 
effect,  this  produced  four  4  by  19  data  arrays  comparable  to  the  data 
arrays  described  above  only  with  respondents  grouped  by  job  classifica¬ 
tion.  Our  statistical  test  was  thus  designed  to  determine  whether  these 
four  groups  of  respondents  were  rating  the  research  areas  independently  or 
with  a  substantial  degree  of  concurrence  on  the  survey. 

Since  the  data  from  the  survey  were  ordinal  data,  a  nonparametric  statis¬ 
tical  test  was  appropriate  for  our  purposes.  There  were  two  nonparametric 
tests  that  were  available  to  us  in  the  SPSS  statistical  software  package 
for  examining  the  independence  of  rankings  hypothesis  discussed  above. 
Both  the  Friedman  two  way  analysis  of  variance  by  ranks  and  the  Kendall 
coefficient  of  concordance  provide  Chi-square  test  statistics  that  suffice 
for  this  purpose.  The  intiutive  explanation  of  the  test  procedures  was  the 
same  in  both  cases  but  the  expression  for  the  calculated  test  statistic 


differs.  We  performed  the  test  using  both  packages  and  were  reassured  to 
discover  that  test  results  were  virtually  Identical. 

Basically,  each  method  first  ranks  the  19  research  areas  for  each  group  of 
respondents  on  each  criterion  (and  for  the  linearly  combined  criteria).  If 
In  fact  the  respondent  rankings  are  Independent,  then  the  likelihood  of  a 
research  area  receiving  the  highest  or  lowest  ranking  from  any  respondent 
group  Is  a  chance  event.  That  is,  the  distribution  of  ranks  in  any  column 

would  be  a  matter  of  chance  and  we  would  expect  the  ranks  of  1,2 . 19  to 

appear  In  all  columns  with  about  equal  frequency.  More  formally,  if  the 
rankings  were  independent  across  the  respondent  groups  (that  is,  no  concur¬ 
rence  among  them),  then  the  ranks  In  each  column  would  represent  a  random 

sample  from  the  discontinuous  rectangular  distribution  of  1 . 19  and  the 

rank  totals  In  each  column  would  be  nearly  equal.  Alternatively,  if  the 
respondent  groups  were  "thinking  alike"  more  so  than  not  alike  in  their 
rankings,  then  the  column  totals  would  differ  from  one  column  to  another. 
Equivalently,  the  mean  rank  from  one  column  (researh  area)  to  another  would 
differ  from  one  to  another. 

The  Friedman  two  way  analysis  of  variance  by  ranks  constructs  a  test  sta¬ 
tistic  that  utilizes  the  column  sums  to  evaluate  independence  of  rankings 
among  the  respondents.  The  larger  the  variation  in  the  rank  totals  (column 
sums)  across  each  research  area,  the  larger  the  Friedman  test  statistic, 
and  the  greater  the  chances  of  rejecting  the  mull  hypothesis  of  indepen¬ 
dence  among  the  respondent  rankings  in  the  four  groups  of  respondents.  The 
Friedman  test  statistic  appears  below: 

12  K 

x2r  =  -  Rj2  -  3N  (K  +  1) 

NK  (K  +  1)  i  =  1 

The  x2r  statistic  has  an  approximate  Chi-square  distribution  with  K  -  1 
degrees  of  freedom.  If  the  calculated  test  exceeds  the  look-up  value  of 
the  Chi-square  variate  from  a  table,  then  we  would  reject  the  null 
hypothesis  that  all  research  areas  (topics)  are  ranked  the  same  at  a 
specified  level  of  significance. 


The  Kendall  coefficient  of  concordance  (W)  Is  a  measure  of  the  degree  of 
agreement  among  the  respondents  that  ranges  between  zero  and  one.  The 
numerator  of  the  measure  contains  the  sum  of  squared  deviations  of  column 
rank  sums  from  the  mean  column  rank  sum  over  the  Items  ranked  (the  19  re¬ 
search  areas  In  this  case).  As  discussed  above,  the  larger  this  measure, 
the  greater  the  agreement  among  the  "judges"  in  their  rankings.  The  deno¬ 
minator  of  W  represents  thw  maximum  value  of  the  numerator  or  the  value 
that  It  would  be  if  there  were  perfect  agreement  among  the  respondents. 
Thus,  the  closer  to  unity  the  Kendall  coefficient  happens  to  be,  the  more 
the  judges  are  In  agreement  in  their  rankings.  The  formula  for  the  Kendall 
coefficient  Is  as  follows: 


S 


1/12  N2  (K^  -  K) 
K 


,  0  <  W  >  1 


K  Rj 

where  S  =  Rj  -  i  =  1 

i  =  l  - 


K  *  number  of  research  areas 
N  =  number  of  respondent  groups 
and  Ri  =  the  rank  sum  of  column  i. 


In  both  the  Friedman  and  Kendall  cases,  ties  among  the  rankings  for  the 
items  ranked  were  each  assigned  to  mean  ranking.  For  example,  if  three 
research  areas  tied  for  the  third  highest  ranking,  each  would  be  assigned  a 
ranking  of  four  in  the  calculations.  A  large  number  of  ties  deflates  the 
value  of  W  so  the  connection  factor  below,  when  deducted  from  the  denomina¬ 
tor  in  the  W  expression,  offsets  this  effect: 

N 

T*  =  N  Ti 

i  =  1 


where  j  =  1 

T  =  _ 

12 
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the  number  of  groups  of  ties  within  the  rankings  of 
the  "ith"  respondent 


ni  = 


t-jj  =  the  number  of  research  areas  in  the  "jth"  group  tied 
for  a  given  rank  by  the  "ith"  respondent. 

To  test  the  significance  of  W,  one  may  determine  the  probability  of  obtain¬ 
ing  of  measure  as  large  as  an  observed  S  on  the  null  hypothesis  that  there 
is  independence  among  the  respondents  in  their  rankings.  The  test  statis¬ 
tic  below  is  approximately  distributed  as  a  Chi-squared  variate  with  K-l 
degrees  of  freedom, 

X2  =  N  {K  -  1)  W 
K-l 

and  may  be  used  to  determine  the  probability  of  obtaining  a  value  as  large 
as  an  observed  W  on  the  null  hypothesis  of  independence  in  respondent  rank¬ 
ings.  If  this  probability  is  very  small,  such  as  less  than  five  or  one 
percent,  then  it  is  safe  to  say  that  the  agreement  observed  among  the 
respondents  is  greater  than  would  occur  by  chance. 

We  calculated  both  the  Friedman  and  Kendall  test  statistics  using  the  SPSS 
statistical  packages.  Table  1  reports  the  results  of  our  tests  and  the 
mean  ranking  of  the  19  research  areas  across  the  four  groups  of  respon¬ 
dents.  Note  that  these  rankings  differ  slightly  from  the  corresponding 
rankings  reported  in  the  next  section.  The  prioritization  presented  in  the 
next  section  gives  equal  weight  to  each  survey  participant  in  calculating 
mean  rankings  across  all  participants.  The  rankings  reported  in  Table  1 
below  give  equal  weight  to  each  participant  group  in  calculating  the  mean 
values.  This  means  that  participants  in  smaller  groups  receive  relatively 
higher  weights  than  participants  in  larger  groups  in  determining  the  pri¬ 
oritization  reported  in  Table  1.  Since  the  Friedman  and  Kendall  results 
were  virtually  identical,  we  report  only  the  latter  test  statistics  in  the 


TABLE  1 

KENDALL  COEFFICIENT  TEST  RESULTS 


Mean  Ranking 
And  Statistics 

|  Objective 

Combined 

"Importance" 

"Need" 

"Use" 

1 

Effective  Managers 
(No.  12) 

No.  12 

No.  5 

No.  9 

2 

Compensation 
(No.  5) 

No.  5 

No.  13 

NO.  4 

3 

Classification 
(No.  9) 

No.  9 

NO.  9 

NO.  12 

4 

New  Technology 
(No.  13) 

No.  4 

No.  3 

No.  17* 

5 

Selection  A  Screening 
(No.  4) 

No.  13 

No.  12 

No.  7* 

6 

Models* 

(No.  3) 

No.  7 

No.  1 

No.  18 

7 

Ma.  keting* 

(No.  7) 

No.  1 

No.  15 

No.  3 

8 

Personnel  Practices 
(No.  15) 

No.  15 

No.  10 

No.  19 

9 

Retain  Minorities 
(No.  18) 

No.  3 

No.  4* 

No.  13 

10 

Recruit  Minorities 
(No.  17) 

No.  10 

No.  lj* 

No.  15 

11 

Insure  Non-Discrimination 
(No.  19) 

NO.  19 

No.  18 

No.  5 

12 

Optimum  Mixes 
(No.  1) 

No.  18 

No.  17 

No.  11 

13 

Incentives 
(No.  10) 

No.  17 

No.  7 

No.  1 

14 

Productivity 
(No.  16) 

NO.  16 

No.  19 

No.  16 

( Continued) 


TABLE  1  (Continued) 


KENDALL  COEFFICIENT  TEST  RESULTS 

Mean  Ranking 
And  Statistics 

Objective 

Combined 

" Importance" 

“Need" 

"Use" 

15 

Reduce  Barriers 
(No.  11) 

No.  6 

No.  14 

No.  10 

16 

Increase  KSA's 
(No.  14) 

No.  14 

No.  6 

No.  6 

17 

Job  Environment 
(No.  6) 

No.  11 

No.  11 

NO.  14 

18 

Career  Development 
(No.  8) 

No.  15 

No.  8 

No.  8 

19 

Optimum  Ratios 
(No.  2) 

No.  2 

No.  2 

No.  2 

Kendall 

Coefficient 

(W) 

.570 

.522 

.560 

.600 

Chi-Square 

Statistic 

41.035 

37.578 

40.356 

43.205 

Level  of 

.001 

.004 

.002 

.001 

Significance 


SOURCE:  Calculated  by  SRA  Technologies,  Inc. 

ROTE:  *  Indicates  a  tie  for  the  ranking. 


The  level  of  significance  Indicates  the  probability  of  achieving 
a  Kendall  coefficient  as  large  as  the  observed  value  on  the  null 
hypothesis  that  there  Is  no  concordance  In  the  respondent's 
rankings. 


The  numbers  associated  with  each  research  area  refer  to  the 
ordering  on  the  survey  questionnaire. 


The  test  results  reported  in  Table  1  demonstrate  that  the  respondent  groups 
tended  to  agree  more  than  disagree  in  their  research  area  rankings,  and  the 
degree  of  concordance  was  statistically  significant.  That  is,  on  the  basis 
of  our  test  results,  the  degree  of  concordance  among  the  respondent  groups 
was  greater  than  would  occur  by  chance.  The  mean  rankings  reported  in 
Table  1  represents  the  best  estimate  of  the  respondent  group  rankings,  but, 
as  we  pointed  out  above,  these  rankings  differ  slightly  from  those  reported 
in  Section  II  in  which  each  individual  receives  the  same  relative  weight  in 
calculating  the  mean  values. 

As  a  final  note,  the  mean  rankings  do  not  necessarily  establish  the  "best" 
prioritization.  They  instead  simply  represent  the  best  estimate  of  the 
commonly  agreed  upon  prioritization  by  the  sample  members  (or  sample  member 
groups). 

Research  Topic  Analysis:  Survey  respondents  were  also  asked  to  check  off 
projects  on  research  topics  that  they  consider  promising  from  among  a  list 
provided  on  the  questionnaire  under  each  research  area.  The  number  of 
topics  listed  under  each  research  area  varied  from  3  to  16  on  the  survey 
Instrument,  so  we  created  19  data  arrays  (for  research  topics  in  each  of  19 
research  areas)  of  36  by  K*  dimensions  where  K*  varied  between  3  and  16 
(research  topics). 

Our  statistical  analysis  focused  on  testing  for  agreement  among  the  respon¬ 
dents  regarding  research  topic  rankings  within  the  14  research  areas  ranked 
the  highest  among  the  19  on  the  questionnaire.  The  rank  order  of  research 
topics  within  these  14  research  areas  was  of  no  particular  interest  beyond 
its  use  in  the  statistical  testing  described  below.  Research  topics  re¬ 
ceiving  "checks"  from  fewer  than  one  third  of  the  respondents  were  not 
given  serious  consideration  as  "promising"  projects.  Section  II  reports 
the  "promising"  projects  according  to  this  criterion  in  the  higher  priority 
research  areas.  Below  we  report  the  results  of  our  statistical  tests  to 
determine  whether  or  not  the  four  groups  of  survey  respondents  were  in 
agreement  in  their  research  project  rankings  within  each  of  the  top  14 
research  areas. 


The  first  step  in  the  test  procedure  was  to  aggregate  the  number  of 
"checks"  given  to  each  project  over  the  respondents  in  each  of  the  four  se¬ 
parate  respondent  groups.  This  produced  fourteen  4  by  K*  data  arrays  which 
comprised  the  data  input  for  the  Friedman  and  Kendall  coefficient  statisti¬ 
cal  tests  discussed  above. 

The  test  results  are  reported  in  Table  2  on  the  following  page.  Contrary 
to  the  research  area  rankings,  we  find  that  in  the  majority  of  cases,  the 
four  respondent  groups  were  not  in  agreement.  In  only  six  of  the  14  re¬ 
search  areas  (Nos.  1,  3,  4,  9,  17,  and  18)  was  the  level  of  significance  in 
the  statistical  test  sufficiently  small  to  suggest  that  the  respondent 
groups  concurred  in  their  rankings  of  the  individual  research  topics  within 
these  areas.  That  is,  the  degree  of  agreement  among  the  respondent  groups 
measured  by  the  Kendall  coefficient  was  greater  than  would  have  occurred  by 
chance.  In  the  other  research  areas  (among  the  top  14),  the  observed  level 
of  agreement  was  not  inconsistent  with  the  null  hypothesis  that  the  respon¬ 
dent  group  rankings  were  inherently  independent.  Since  there  was  some 
slight  differences  between  the  Friedman  and  Kendall  test  resuts,  we  report 
both  sets  of  results  in  the  table. 

These  test  results  justify  our  treatment  of  the  research  topic  prioritiza¬ 
tion  discussed  in  Section  II.  Since  In  the  majority  of  cases  the  respon¬ 
dents  did  not  reach  agreement  (in  a  statistical  sense)  on  a  ranking  of  pro¬ 
jects  within  the  research  areas,  we  do  not  report  the  mean  rankings  of  pro¬ 
jects  that  did  result  from  the  survey.  Instead  we  present  all  but  the  pro¬ 
jects  receiving  less  than  13  check  marks  as  "promising"  research  topics 
under  the  higher  priority  research  areas,  and  the  mid-ran ked  research 
areas. 


STAGE  TOO:  FEASIBILITY  MiALYSIS 

The  statistical  analysis  placed  the  research  areas  into  priority  categories 
based  on  relative  importance,  and  Indicated  which  specific  research  pro¬ 
jects  the  respondents  judged  to  be  most  important  within  each  research 
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STATISTICAL  TESTS  FOR  RESPONDENT  GROUP  AGREEMENT 

IN  RANKING  RESEARCH  TOPICS  WTHIN  RESEARCH  AREAS  i 


Kendall  Test 

Friedman  Test 

Research 

Area 

Kendal 1 
Coefficient 
(W) 

Chi 

Square 

Value 

Level 
of  Signi¬ 
ficance 

Chi 

Square 

Value 

Level 
of  Signi¬ 
ficance 

Degree 
of  Freedc 
(K*-l) 

Effective  Managers 
(No.  12) 

.140 

1.68 

.642 

1.42 

.700 

3 

Compensation 
(No.  5) 

.363 

11.62 

.169 

11.13 

.194 

8 

Classification 
(No.  9) 

.605 

14.53 

.024 

12.91 

.044 

6 

New  Technology 
(No.  13) 

.417 

6.67 

.155 

4.75 

.314 

4 

Selection  &  Screening 
(No.  4) 

.737 

8.84 

.031 

8.40 

.038 

3 

S 

Model s 
(No.  3) 

.729 

8.75 

.033 

7.87 

.049 

3 

Marketing 
(No.  7) 

.219 

4.39 

.495 

3.57 

% 

.613 

5 

.J 

Personnel  Practices 
(No.  15) 

.331 

3.97 

.265 

3.37 

.337 

3 

Retain  Minorities 
(No.  18) 

.524 

10.47 

.063 

8.68 

.123 

5 

t 

Recruit  Minorities 
(No.  17) 

.586 

18.76 

.016 

15.87 

.044 

8 

Insure  Non-Discrimination 
(No.  19) 

.071 

.57 

.751 

.50 

.779 

2 

(Continues 


C-10 


TABLE  2  (Continued) 

STATISTICAL  TESTS  FOR  RESPONDENT  GROUP  AGREEMENT 
IN  RANKING  RESEARCH  TOPICS  WITHIN  RESEARCH  AREAS 


Kendall  Test 

Friedman  Test 

Research 

Area 

Kendall 

Coefficient 

(W) 

Hi 

HI 

Level 
of  Signi¬ 
ficance 

Chi 

Square 

Value 

Level 
of  Signi¬ 
ficance 

Degrees 
of  Freedom 
(K*-l) 

Optimum  Mixes 

.770 

9.24 

.026 

8.77 

.032 

3 

(No.  1) 

Incentives 

.107 

1.71 

.790 

1.45 

.836 

4 

(No.  10) 

Productivity 

.090 

1.08 

.781 

.97 

.807 

3 

(No.  16) 


SOURCE:  Calculated  by  SRA  Technologies,  Inc. 

NOTE:  The  level  of  significance  Indicates  the  probability  of  achieving  a  test  statistic  as 

large  as  the  observed  value  of  the  null  hypothesis  that  three  Is  no  agreement  among 
the  respondents  In  their  research  project  rankings. 

The  numbers  associated  with  each  research  area  refer  to  the  ordering  on  the  survey 
questionnaire.  See  the  next  section  for  a  more  complete  discussion  of  the  research 
areas  and  projects. 


area.  The  next  stage  consisted  of  a  series  of  technical  feasibility  judge¬ 
ments.  The  factors  considered  In  this  process  Included: 

o  The  adequacy  of  existing  knowledge  In  an  area,  based  upon  the 
literature  review  In  the  Roadmap. 

o  The  likelihood  of  research  success; 

o  The  time  required  to  complete  the  research;  and 

o  The  estimated  cost  to  complete  the  research. 

For  each  of  the  four  highest  ranked  research  areas,  a  sequenced  array  of 
potential  research  projects  was  constructed.  For  the  mid-ranked  areas,  the 
projects  seen  as  most  Important  by  questionnaire  respondents  were  noted. 
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